Method and system for recalibrating sensing devices without familiar targets

ABSTRACT

Methods and systems for recalibrating sensing devices without using familiar targets are provided herein. The method may include: capturing a scene using a sensing device; determining whether or not the device is calibrated; in a case that said sensing device is calibrated, adding at least one new landmark to the known landmarks; in a case that said sensing device is not calibrated, determining at least some of the captured objects as objects stored on a database as known landmarks at the scene whose position is known; and calibrating the sensing device based on the known landmarks. The system may have various architectures that include a sensing device which captures images of the scene and further derive 3D data on the scene of some form.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Ser. No. 14/872,158, filedon Oct. 1, 2015, issued as U.S. Pat. No. 9,681,118 on Jun. 13, 2017,which is incorporated in its entirety herein by reference.

FIELD OF THE INVENTION

The present invention relates generally to the field of calibratingsensing devices, and more particularly to such calibration carried outwithout the need for predefined calibration targets.

BACKGROUND OF THE INVENTION

Prior to setting forth the background of the invention, it may behelpful to set forth definitions of certain terms that will be usedhereinafter.

The term ‘camera’ or ‘sensing device” as used herein is broadly definedas any combination of one or more sensors that are configured to capturethree-dimensional data of a scene, directly or through furthermanipulation. An exemplary sensing device may include a pair of cameraswhich are configured to capture passive stereo which may derive depthdata by comparing the images taken from different locations. Anotherexample for a sensing device may include a structured light sensor whichis configured to receive and analyze reflections of a predefined lightpattern that has been projected onto the scene. A structured lightsystem includes a radiation source (such as a laser source) and a sensorconfigured to capture the reflections from the radiation source Yetanother example would be a single camera for capturing 2D imagestogether with a measurement device that can measure the distance thesensing device travels. Such a measurement device can be, but notlimited to, an inertial measurement unit (IMU) or odometer.

The term ‘three dimensional rig’ or ‘3D Rig’ as used herein is definedas any device for mounting at least two cameras or at least one cameraand a radiation source that together form a 3D-system capable ofcapturing images and videos of a scene. A 3D Rig must provide thepossibility to mount two cameras or one camera and one radiation source,with a horizontal or vertical offset and adjust the cameras in allpossible axes.

The term ‘calibration’ and more specifically “camera calibration” asused herein is defined as the process of adjusting predefined parametersso that the camera, or the sensing device may operate optimally.Calibration of a 3D Rig involves the calibration of a plurality ofcameras. The calibration of a 3D Rig includes the alignment orconfiguration of several cameras and in particular the process ofidentifying the deviation of several parameters relating cameras'spatial alignment from a predefined value tolerance and remedying theidentified deviation. It is said that a certain sensing device iscalibrated whenever the estimated calibration parameters, reflect theactual calibration parameters, taken at the specified timeslot, withinan agreeable margin. It is possible to verify that a device iscalibrated by comparing distances measured by the device with distancesobtained from un-related sources (e.g., directly measured).

The term “landmark” as used herein relates to visual features used by avariety of computer vision applications such as image registration,camera calibration, and object recognition. Using landmarks isadvantageous as it offers robustness with regard to lightning conditionsas well as the ability to cope with large displacements in registration.As defined herein, a landmark comprises both artificial and naturallandmarks. Exemplary landmarks may include corners or repetitivepatterns in images.

One of the challenges of sensing devices that are required to capture ascene in real time, such as wearable near-eye displays, is to maintaincalibration in real-time. More specifically, as such devices tend tolose its initial calibration quite easily (e.g., due to physical impactapplied to the sensing device), it is essential to be able to regaincalibration quickly without requiring manual intervention.

While calibrating a sensing device using pre-registered landmarks iswell known in the art, when operating in an unfamiliar scene in whichpre-registered landmarks cannot be identified, calibration becomes amore difficult task.

It would be, therefore, advantageous to provide a method and a systemthat addresses the calibration challenge in unfamiliar scenes.

SUMMARY OF THE INVENTION

Some embodiments of the present invention provide a method and a systemthat enable recalibrating of sensing devices without familiar targets,so that newly generated landmarks may be useful in a future calibrationprocess of the sensing device.

According to some embodiments, the method may include: capturing a sceneusing a sensing device; determining whether or not the device iscalibrated; in a case that said sensing device is calibrated, adding atleast one new landmark to the known landmarks; in a case that saidsensing device is not calibrated, calibrating the sensing device basedon the known landmarks. The system may have various architectures thatinclude a sensing device which captures images of the scene and furtherderive 3D data on the scene of some form.

According to another non-limiting embodiment, the method may include:capturing a scene using a sensing device; determining whether thecaptured scene contains a minimal number of known landmarks sufficientfor determining whether the sensing device is calibrated; in a case thatthe captured scene contains said minimal number of landmarks, checkingwhether said sensing device is calibrated; in a case that said sensingdevice is calibrated, adding at least one new landmark to the knownlandmarks; in a case that said sensing device is not calibrated,calibrating the sensing device based on the known landmarks; and in acase that the captured scene does not contain said minimal number oflandmarks, checking calibration of the sensing device without usingknown landmarks, and in a case that the sensing device is calibrated,adding at least one new landmark to the known landmarks.

It should be noted that determining whether a sensing device or a 3D rigis fully calibrated requires fewer landmarks than the actual calibrationprocess. Therefore, in case that a calibrated condition is identified,it would be easier to generate more landmarks of the unfamiliar scene.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, however, both as to organization and method of operation,together with objects, features, and advantages thereof, may best beunderstood by reference to the following detailed description when readwith the accompanying drawings in which:

FIG. 1 is a block diagram illustrating non-limiting exemplaryarchitectures of a system in accordance with some embodiments of thepresent invention;

FIG. 2A is a high level flowchart illustrating non-limiting exemplarymethod in accordance with some embodiments of the present invention;

FIG. 2B is a high level flowchart illustrating another non-limitingexemplary method in accordance with some embodiments of the presentinvention; and

FIG. 3 is a block diagram illustrating yet another aspect in accordancewith some embodiments of the present invention.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numerals may be repeated among the figures toindicate corresponding or analogous elements.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, various aspects of the present inventionwill be described. For purposes of explanation, specific configurationsand details are set forth in order to provide a thorough understandingof the present invention. However, it will also be apparent to oneskilled in the art that the present invention may be practiced withoutthe specific details presented herein. Furthermore, well known featuresmay be omitted or simplified in order not to obscure the presentinvention.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “processing,” “computing,”“calculating,” “determining,” or the like, refer to the action and/orprocesses of a computer or computing system, or similar electroniccomputing device, that manipulates and/or transforms data represented asphysical, such as electronic, quantities within the computing system'sregisters and/or memories into other data similarly represented asphysical quantities within the computing system's memories, registers orother such information storage, transmission or display devices.

FIG. 1 is a block diagram illustrating an exemplary architecture onwhich embodiments of the present invention may be implemented. System100 may include a sensing device 110 configured to capture a scene.System 100 may further include a landmarks database 120 configured tostore landmarks known in the scene (e.g., 10A, 10B, and 10C).

In some embodiments, system 100 may further include a computer processor130 configured to determine whether or not the device is calibrated,based on the captured image, wherein in a case that said sensing deviceis calibrated, the computer processor configured to add at least one newlandmark to the known landmarks, and wherein in a case that said sensingdevice is not calibrated, the computer processor is configured tocalibrate the sensing device based on the known landmarks.

In other embodiments, system 100 may further include a computerprocessor 130 configured to determine whether the captured scenecontains a minimal number of known landmarks sufficient for determiningwhether the sensing device is calibrated. In some embodiments, computerprocessor 130 is further configured to carry out the calibrationprocess, whereas in other embodiments, a different computer processor isbeing used for the actual calibration process.

It should be noted that the number of landmarks sufficient fordetermining whether the sensing device is calibrated is usually lowerthan the number of landmarks sufficient for actually calibrating thesensing device. This important property is used herein as the ability todetermine whether the sensing device is calibrated.

In a case that the captured scene contains said minimal number oflandmarks, the computer processor is configured to check whether sensingdevice 110 is calibrated, wherein in a case that sensing device 110 iscalibrated, computer processor 130 is configured to add at least one newlandmark 132 to landmarks database 120, wherein in a case that sensingdevice 110 is not calibrated, computer processor 130 is configured tocalibrate sensing device 110 based on the known landmarks, and whereinin a case that the captured scene does not contain the minimal number oflandmarks, computer processor 130 is configured to check calibration ofsensing device 110 without using known landmarks (e.g., 10A, 10B, and10C), and in a case that sensing device 110 is calibrated, computerprocessor 130 is configured to add at least one new landmark 132 to thelandmarks database 120.

FIG. 2A is a high level flowchart illustrating a method 200A forrecalibrating sensing devices without familiar targets. The method mayinclude: capturing a scene using a sensing device 210; determiningwhether the sensing device is calibrated 230; in a case that the sensingdevice is calibrated, adding at least one new landmark to the knownlandmarks 250. In a case that said sensing device is not calibrated,determining at least some of the captured objects as objects stored on adatabase as known landmarks at the scene whose relative position isknown 235 and calibrating the sensing device based on the knownlandmarks 290. In some embodiments, in a case that the sensing device isdetermined as non-calibrated it is first checked whether there is asufficient number of landmarks for calibrating the sensing device 240and only in a case that the captured scene contains the minimal numberof landmarks, calibrating the sensing device based on the knownlandmarks, Otherwise, the process ends 280.

FIG. 2B is a high level flowchart illustrating a method 200B forrecalibrating sensing devices without familiar targets. The method mayinclude: capturing a scene using a sensing device 210; determiningwhether the captured scene contains a minimal number of known landmarkssufficient for determining whether the sensing device is calibrated 220;in a case that the captured scene contains the minimal number oflandmarks, checking whether said sensing device is calibrated 230, in acase that said sensing device is calibrated, adding at least one newlandmark to the known landmarks 250; in a case that said sensing deviceis not calibrated, calibrate the sensing device based on the knownlandmarks 290; and in a case that the captured scene does not containsaid minimal number of landmarks, checking calibration of the sensingdevice without using known landmarks 260, and in a case that the sensingdevice is calibrated, adding at least one landmark to the knownlandmarks 250, Otherwise, the process ends 270. In some embodiments, ina case that the sensing device is determined as non-calibrated, it isfirst checked whether there is a sufficient number of landmarks forcalibrating the sensing device 240 and only in a case that the capturedscene contains the minimal number of landmarks, calibrating the sensingdevice based on the known landmarks, Otherwise, the process ends 280.

In accordance with some embodiments of the present invention, two modes(usually both are applied) are operable for identifying whether thesensing device is calibrated. In the first one, without the “learned”targets matching features are determined between the components of thesensing device and check if they meet the geometric requirements (suchas epipolar lines). One limitation that is present there is that wecannot recognize errors that are parallel to the epipolar lines (e.g.,in the case of two cameras).

In the second mode, with learned targets, it is determined which of thelearned targets are visible, possibly using standard target recognition.For each of the learned targets: detect the features from the databasein the current frame from the 3D rig, with high-accuracy, while not allfeatures need to be found. Then, it is checked how accurately the knownfeatures comply with the sensing device's geometric model and itscurrent parameters. If the accuracy is worse than some threshold, thenthe sensing device is deemed not calibrated.

According to some embodiments of the present invention, the check ofcalibration is performed using the known landmarks.

According to some embodiments of the present invention, the at least onenew landmark is selected based on predefined criteria including at leastone of: textured area, uniqueness of appearance; low-complexitystructure; size and distribution of the points; and distance from thesensing device.

According to some embodiments of the present invention, the method mayfurther include a step of maintaining a landmarks database configured tostore known and newly added landmarks, wherein the database issearchable. The database may be configured to further store lowresolution versions or other descriptors of the landmarks, for quickretrieval.

According to some embodiments of the present invention, searching thedatabase for a known landmark takes into account various points ofviews.

According to some embodiments of the present invention, the newlandmarks are stored together with metadata (e.g., descriptors or hashfunctions) for facilitating contextual search of said landmarks.

According to some embodiments of the present invention, the furtheridentified landmarks are stored in a way to allow fast searching throughthe database, wherein landmarks from the same environment are clustered.

FIG. 3 is a block diagram illustrating yet another aspect in accordancewith some embodiments of the present invention. An exemplaryimplementation of the algorithm in accordance with some embodiments ofthe present invention is described herein. Capturing device 110 providesdata relating to 2D points in the frame 112 while landmarks databaseprovide data relating to the landmark 122 in the scene. The landmarkdata 122 are then being input to a formula 310 that is tailored for thespecific 3D rig or sensing device. The formula then maps the landmarkdata 122 to a calculated 2D point or area on the frame 320 which is thencompared by a comparator 330 with the data relating to 2D points in thecaptured frame 112 to yield a delta 340 being the difference between thecalculated 2D point on the frame and the data on the 2D point in theframe. Then, in order to minimize delta 340, position 312, angles 314and calibration parameters 316 are being adjusted in various mannersknown in the art so as the delta 340 is minimal. Once delta 340 isminimized, the 3D rig or sensing device is calibrated and the adjustedcalibration parameters are derived and stored.

According to some embodiments of the present invention, the calibrationis solved by minimizing an objective function measuring the projectionerrors. Alternatively, it can be solved by minimizing by using gradientdescent. Further alternatively, the minimizing is carried out by usingthe Levenberg-Marquardt optimization technique.

Advantageously, some embodiments of the present invention are notlimited to a 3D rig or a sensing device that captures 3D images. It mayalso be used to calibrate a system that includes a 2D sensing devicecoupled to an inertial measurement unit (IMU). In such a system, thegeometric constraints are given between different frames of the samecamera, with the geometery being measured from the IMU. Alternatively,the sensing device is a 2D camera mounted on a moving platform such as arobot, from which the motion can be measured from wheel angles and wheelrotations. It should be understood that the aforementioned embodimentsshould not be regarded as limiting and further applications may beenvisoned in order to adress further use cases.

In the above description, an embodiment is an example or implementationof the inventions. The various appearances of “one embodiment,” “anembodiment” or “some embodiments” do not necessarily all refer to thesame embodiments.

Although various features of the invention may be described in thecontext of a single embodiment, the features may also be providedseparately or in any suitable combination. Conversely, although theinvention may be described herein in the context of separate embodimentsfor clarity, the invention may also be implemented in a singleembodiment.

Reference in the specification to “some embodiments”, “an embodiment”,“one embodiment” or “other embodiments” means that a particular feature,structure, or characteristic described in connection with theembodiments is included in at least some embodiments, but notnecessarily all embodiments, of the inventions.

It is to be understood that the phraseology and terminology employedherein is not to be construed as limiting and are for descriptivepurpose only.

The principles and uses of the teachings of the present invention may bebetter understood with reference to the accompanying description,figures and examples.

It is to be understood that the details set forth herein do not construea limitation to an application of the invention.

Furthermore, it is to be understood that the invention can be carriedout or practiced in various ways and that the invention can beimplemented in embodiments other than the ones outlined in thedescription above.

It is to be understood that the terms “including”, “comprising”,“consisting” and grammatical variants thereof do not preclude theaddition of one or more components, features, steps, or integers orgroups thereof and that the terms are to be construed as specifyingcomponents, features, steps or integers.

If the specification or claims refer to “an additional” element, thatdoes not preclude there being more than one of the additional element.

It is to be understood that where the claims or specification refer to“a” or “an” element, such reference is not be construed that there isonly one of that element.

It is to be understood that where the specification states that acomponent, feature, structure, or characteristic “may”, “might”, “can”or “could” be included, that particular component, feature, structure,or characteristic is not required to be included.

Where applicable, although state diagrams, flow diagrams or both may beused to describe embodiments, the invention is not limited to thosediagrams or to the corresponding descriptions. For example, flow neednot move through each illustrated box or state, or in exactly the sameorder as illustrated and described.

Methods of the present invention may be implemented by performing orcompleting manually, automatically, or a combination thereof, selectedsteps or tasks.

The descriptions, examples, methods and materials presented in theclaims and the specification are not to be construed as limiting butrather as illustrative only.

Meanings of technical and scientific terms used herein are to becommonly understood as by one of ordinary skill in the art to which theinvention belongs, unless otherwise defined.

The present invention may be implemented in the testing or practice withmethods and materials equivalent or similar to those described herein.

While the invention has been described with respect to a limited numberof embodiments, these should not be construed as limitations on thescope of the invention, but rather as exemplifications of some of thepreferred embodiments. Other possible variations, modifications, andapplications are also within the scope of the invention. Accordingly,the scope of the invention should not be limited by what has thus farbeen described, but by the appended claims and their legal equivalents.

1. A method comprising: providing captured data relating to one or more captured 2D points in a frame of a scene captured by a sensing device; providing landmark data relating to a landmark in the scene from a landmark database; mapping the landmark data to one or more calculated 2D points in a frame of the scene; comparing the captured 2D points in the frame with the data relating to the calculated 2D points in the frame to yield a delta being a difference between the data relating to the captured 2D points and the calculated 2D points; minimizing the delta by adjusting one or more calibration parameters; and calibrating the sensing device based on the adjusted calibration parameters.
 2. The method of claim 1 comprising minimizing the delta by minimizing an objective function measuring projection errors.
 3. The method of claim 1 comprising minimizing the delta by using gradient descent.
 4. The method of claim 1 comprising minimizing the delta by using the Levenberg-Marquardt optimization technique.
 5. The method of claim 1, wherein the captured data is captured by a 3D sensing device that captures 3D images.
 6. The method of claim 1, wherein the captured data is captured by a 2D sensing device coupled to an inertial measurement unit (IMU).
 7. The method of claim 1, wherein the captured data is captured by a 2D sensing device mounted on a moving platform.
 8. A system comprising: a sensing device configured to provide captured data relating to one or more captured 2D points in a frame of a scene captured; a database configured to provide landmark data relating to a landmark in the scene; and a computer processor configured to: map the landmark data to one or more calculated 2D points in a frame of the scene, compare the captured 2D points in the frame with the data relating to the calculated 2D points in the frame to yield a delta being a difference between the data relating to the captured 2D points and the calculated 2D points, minimize the delta by adjusting one or more calibration parameters, and calibrate the sensing device based on the adjusted calibration parameters.
 9. The system of claim 8, wherein the computer processor is configured to minimize the delta by minimizing an objective function measuring projection errors.
 10. The system of claim 8, wherein the computer processor is configured to minimize the delta by using gradient descent.
 11. The system of claim 8, wherein the computer processor is configured to minimize the delta by using the Levenberg-Marquardt optimization technique.
 12. The system of claim 8, wherein the sensing device is a 3D sensing device that captures 3D images.
 13. The system of claim 8, wherein the sensing device is a 2D sensing device coupled to an inertial measurement unit (IMU).
 14. The system of claim 8, wherein the sensing device is a 2D sensing device mounted on a moving platform. 